Image sensors, also known as imagers, were developed in the late 1960s and early 1970s primarily for television image acquisition, transmission, and display. An imager absorbs incident radiation of a particular wavelength (such as optical photons, x-rays, or the like) and generates an electrical signal corresponding to the absorbed radiation. There are a number of different types of semiconductor-based imagers, including charge coupled devices (CCDs), photodiode arrays, charge injection devices (CIDs), hybrid focal plan arrays, and CMOS imagers.
These imagers typically consist of an array of pixel cells containing photosensors, where each pixel produces a signal corresponding to the intensity of light impinging on that element when an image is focused on the array. These signals may then be stored, for example, to display a corresponding image on a monitor or otherwise used to provide information about the optical image. The photosensors are typically phototransistors, photoconductors or photodiodes. The magnitude of the signal produced by each pixel, therefore, is proportional to the amount of light impinging on the photosensor.
To allow the photosensors to capture a color image, the photosensors must be able to separately detect, for example when using a Bayer pattern, red (R) photons, green (G) photons and blue (B) photons. Accordingly, each pixel must be sensitive only to one color or spectral band. For this, a color filter array (CFA) is typically placed in front of the pixels so that each pixel measures the light of the color of its associated filter. Thus, each pixel of a color image sensor is covered with either a red, green or blue filter, according to a specific pattern.
For most low cost CMOS or CCD image sensors, the color filters are integrated with the sensor cells. A common example of a color filter pattern is the tiled color filter array illustrated in U.S. Pat. No. 3,971,065, (the disclosure of which is incorporated by reference herein) and commonly referred to as “the Bayer pattern” color filter.
As shown in FIG. 1, the Bayer pattern 100 is an array of repeating red (R), green (G), and blue (B) filters. In the Bayer pattern 100, red, green and blue pixels are arranged so that alternating red and green pixels are on a first row 105 of an array, and alternating blue and green pixels are on a next row 110. These alternating rows are repeated throughout the array. Thus, when the image sensor is read out, line by line, the pixel sequence for the first line reads GRGRGR etc., and then the alternate line sequence reads BGBGBG etc. This output is called sequential RGB or sRGB.
In the Bayer pattern 100, sampling rates for all three basic colors are adjusted according to the acuity of the human visual system. That is, green color, to which the human eye is most sensitive and responsive, is sensed with a larger number of sensors, whereas blue and red color, for which the human vision has less resolution, are sensed with a fewer number of sensors. This is why in the Bayer pattern, the green-sensitive elements occur at every other array position, while the red-sensitive elements and the blue-sensitive elements occur at every fourth array position.
As shown in FIG. 2 in a solid state image sensor, the Bayer patterned filters may be formed over an array 200 of pixel sensor cells 205. Specifically, an array 200 of pixel sensor cells 205 is formed on a semiconductor substrate 210. Each pixel sensor cell 205 has a photosensitive element 215, which may be any photon-to-charge converting device, such as a photogate, photoconductor or photodiode. The color filter array 220 is typically formed over a metal layer 225 in the array 200, separated from the photosensors 215 by various metallization and insulating layers such as an interlevel dielectric layer (ILD) 235 containing insulator metallization patterns and a passivation layer 230. The metal layer 225 may be opaque and used to shield the areas of the pixels that are not light sensitive. Convex lenses 240 are formed over the color filters 220. In operation, incident light is focused by the lenses 240 through the filters 220 to the photosensitive element 215.
For a Bayer pattern filter, values for red, green and blue are necessary for each pixel. Since each pixel sensor cell is only sensing one color, values for the remaining two colors are calculated by interpolation from the neighboring pixel cells that are sensing the missing colors. This color plane interpolation is known as demosaicing. For example, with reference to FIG. 1, pixel sensor cell 115 is associated with a green filter, which causes pixel sensor cell 115 to sense green light and produce a signal which represents only green light. In order to obtain an approximation of the amount of red and blue light for pixel sensor cell 115, a value may be interpolated from the neighboring red pixel sensor cells 120 and 125 and the neighboring blue pixel sensor cells 130 and 135, respectively. If demosaicing is not performed correctly, the resulting image may suffer from the inclusion of highly visible color artifacts.
The article entitled “Color Plane Interpolation Using Alternating Projections” published in the IEEE Transactions on Image Processing, Vol. II, No. 9 in September 2002 and written by Bahadir K. Gunturk, Yucel Altunbasak and Russell M. Mersereau (the disclosure of which is incorporated by reference herein) compares several demosaicing techniques. As described each of these demosaicing techniques have their respective advantages and disadvantages.
As described, for each pixel sensor cell, the value used for a first color is based on a sensed color and the values for the remaining two colors are based on an interpolated value from the sensed values of corresponding neighboring pixels. Each of the sensed values are representative of the color value at the center of the pixel. Each of the interpolated color values are also representative of the value at the center of the pixel. The interpolated signal is inherently of a lower quality than the originally sensed signal. For example, a interpolated red color value at the center of a pixel would be different than a sensed red color value for the center of the same pixel. These differences in quality may be amplified by the sharpening stage commonly incorporated in digital camera systems or other post capture image adjustment programs. Therefore the mixture of sensed color values and interpolated color values causes in-consistent quality across the picture. A few specific examples of the artifacts are described in the following paragraph.
Demosaicing methods which only reconstruct the missing color components may result in artifacts such as so-called zipper effects and random color dots. Zipper effects may be caused by variations in interpolation and may result in a line of dots along horizontal or vertical lines in regular intervals, for example, one dot every two pixels. This may result from the variations in the separate interpolations of the red and blue values causing the zipper effect in the final image. Random color dots appear mostly along edges within the image and at the end points of lines when lighter area gets light dots and darker area gets dark dots. This effect is most likely caused by efforts during interpolation to produce sharp edges, and the use of other nearby pixels in determining color values. In addition to zipper effects and random color dots, additional unbalance problems result when the two green samples in a particular Bayer mosaic (the elementary 4 cell square) are digitized by different pixel processing channels.
Accordingly, there is a desire and need for color plane interpolation that mitigates the disadvantages of known demosaicing techniques.